1. Field of the Disclosure
This disclosure relates to a method for inspecting objects in 3-dimensional vision, including inspecting electronic components which features' texture may require inspection that is detailed to a sub-pixel scale. An apparatus implementing such method is also disclosed.
2. Discussion of the Background Art
In a high-precision manufacturing environment where high volume of precision small objects are made, such as microelectronics fabrication and packaging of components, it is often necessary for each of the completed object such as a microelectronic package to be inspected for quality control purposes. Due to the large volume, such inspection has to be automated and due to the inspection involving examining specific critical feature of the package, such as co-planarity of the contact elements in the form of balls in a ball grid array (BGA) device, 3-dimensional vision is required.
Some vision systems use a single axial camera with a plurality of reflective surfaces to pick up different views of the object to be integrated in a single image such as that disclosed in U.S. Pat. No. 6,055,054 (Beaty, et al.), WO01/04567 and US-2002/37098. Normally, an algorithm in the form of software is used to process the different reflected views of the object to be integrated into a single image to be inspected.
Some other vision systems use at least two cameras such as that disclosed in U.S. Pat. No. 6,778,282 (ICOS Vision Systems), U.S. Pat. No. 6,064,756 and U.S. Pat. No. 6,064,757 (both by Beaty, et al.). These vision systems generally employ a first camera to capture an incident view of the object and a second camera to take a second view from another angle either directly or indirectly via reflective surfaces.
The present method for 3-dimensional vision inspection makes use of certain functions of Halcon™ Imaging Processing Library (version 7.1) by MVTec Software GmbH of München, Germany. Halcon™ user manual on 3-dimensional vision system entitled Machine Vision in World Coordinates which, along with other relevant Halcon™ publications, are incorporated herein by reference.
The stereoscopic camera system comprises two cameras trained on the same object from different positions. With a pair of cameras, a pair of image points from the projection of a point on the object may be captured. The image points are often referred to as “conjugate points” or “homologous points”. By making use of interior parameters of both cameras and the relative pose of the second camera with respect to the first camera obtained during a prior calibration process, the distance of the said object point from the stereoscopic camera system can be determined.
A simplified configuration of two parallel-trained cameras with identical interior parameters may be shown in FIG. 1 (Prior Art). The basis, i.e. the straight line connecting the two optical centres of the two cameras, may be assumed to be coincident with the x-axis of the first camera. Then, the image to coordinates, u1 and u2, of the projections of the point P (xc, zc) into the image planes may be expressed as follows:
            u      1        =          f      ⁢                        x          c                          z          c                                u      2        =          f      ⁢                                    x            c                    -          b                          z          c                    where f is the focal length and b the length of the basis. The difference between the two image locations of the conjugate points is called the disparity d, which may be calculated as follows:
  d  =            (                        u          1                -                  u          2                    )        =                  f        ·        b                    z        c            
Given the camera parameters and the image coordinates of two conjugate points, the zc coordinate of the corresponding object point P, i.e. the distance from the stereo camera system, can be computed from the following equation:
      z    c    =            f      ·      b        d  
It should be noted that the interior camera parameters of both cameras and the relative pose of the second camera in relation to the first camera are necessary to determine the distance of P from the stereo camera system.
Two basic operations that need to be solved for conventional stereo vision are:
(1)—Determination of the interior parameters of both cameras and the relative pose between them during initial set-up, and
(2)—Determination of a pair of conjugate points for each object point of interest in stereo image pair captured during inspection to compute the distance of said object points from the stereoscopic camera system.
The first operation involves calibration of the stereo camera system whereby a calibration plate is placed such that it is completely within both of the field of views of the stereo cameras. With Halcon™ Imaging Processing Library, the calibration of both images may be carried out simultaneously with the operator termed as “binocular calibration” [see section 7.3, page 94 of Machine Vision in World Coordinates, Edition 2, Halcon™ 7.1.0, published July 2005]. After a successful calibration, a virtual stereo camera system called rectified stereo camera system, of which the rectified coordinate system is with respect to the first camera, will be constructed.
As the accuracy of the 3-dimensional image representation is influenced by the manner in which the cameras are placed, it is important that neither the interior camera parameters (e.g., the focal length) nor the relative pose (e.g., the distance and orientation between the two cameras) of the two cameras should change during the calibration process or between the calibration process, as well as during the ensuing application of the calibrated stereo camera system. It is therefore advisable to mount the two cameras on to a stable platform.
The second operation involves what is called “stereo matching process”, which in Halcon™ processing involves invoking the operators “binocular disparity” or “binocular distance” [see section 7.4, page 97, op. cit.] which process the necessary calculations to obtain world coordinates from the stereo images. In HaIcon™ processing, two image points from a stereo image pair captured from the same object point are referred to as conjugate points. From the principle of stereovision, the difference between image locations of the conjugate points, which is called the “disparity”, can be used to compute the distance of an object point from the rectified stereo camera system.
The Halcon™ Library provides individual functions to compute disparity or distance of the entire overlapping region of the stereo image pair. These functions must first carry out matching operation to establish conjugate points for all image points within the overlapping region of stereo image pair before disparity or distance profile of the region can be computed. The stereo image pair must first be rectified, based on information gathered from calibration, to align their overall Y image coordinate. Thus the ensuing matching process is mainly to find out the difference of conjugate points in X image coordinate. There are 3 different methods, namely “summed absolute difference”, “summed squared differences” and “normalized cross correlation”, available for the matching function and all 3 are based on comparison of gray-level pattern within a small matching window. To obtain accurate results, the surface of the object must have enough textural information to be captured.
Unfortunately, this approach is not useful when determining the distance to a small feature having varying surface curvature (hence varying distance) such as a contact solder ball tip of a ball grid array semiconductor package. Using Halcon™ matching methods would be cumbersome and inaccurate as the ball surface is void of textural features and is therefore not suitable for Halcon™ matching methods.
Nevertheless, Halcon™ methodology also provides functions to compute distance from disparity of conjugate points or distance from the conjugate points directly whereby the conjugate points are to be determined by the user. It should be noted that conjugate points can be taken as rectified image coordinates or original image coordinates.
We are now able to detect the conjugate points of the ball tips using the present method and yet be able to compute the distance of every ball tip from rectified stereo cameras using Halcon functions. The process may be more readily understood from the following disclosure.